, 2013 Lecture 12 : January 5

In the previous lecture we saw the algorithm that builds a decision tree based on a sample. The decision tree is build until zero training error. As we saw before, our goal is to minimize the testing error and not the training error. In order to minimize the testing error, we have two basic options. The first option is to decide to do early stopping. Namely, stop building the decision tree at some point. Different alternatives to perform the early stopping are: (1) when the number of examples in a node is small, (2) The reduction in the slitting criteria is small, (3) a bound on the number of nodes, etc. The alternative approach is to first build a large decision tree, until there is no training error, and then prune the decision. The net effect is the same. In both cases we end with a small decision tree. The major difference is what we observe on the way. When we first build a large tree, we may discover some important sub-structure that in case we stop early, we might miss. In some sense, we can always simulate the early stopping when we first build the large tree and then prune, but definitely cannot do the reverse.